Pairwise comparison techniques for preference elicitation: using test-retest reliability as a quality indicator

This study aims at augmenting the profile distance method (PDM) with techniques that support the elicitation of a relative weight vector. Therefore, prominent methods from the field of preference elicitation and related research are investigated according to their task fitness. Preference elicitation is widely regarded as one integral part of decision support. It has found broad attention in multiple scientific fields, such as psychology of choice or behavioural decision making. The technique of pairwise comparison, often utilized in applied decision support, is examined according to practicability and validity for the estimation of a relative weight vector. The concrete task of weight vector assessment is tested in an experimental setting using widely accepted scales and techniques derived from the literature survey conducted. We distinctly identify two key figures for the measurement of outcome quality, accounting for both mathematical consistency and internal (or human) consistency. Preliminary experimental results from a web-based study with sixty two (62) distinct users provide for valuable insights in consistency ratio and test-retest reliability, indicating that unmodified pairwise comparisons are a suboptimal method for criteria preference elicitation. Additionally, we propose a number of improvements to practical preference elicitation, such as the use of a guided, process-based weight elicitation process.